Undergraduate Course
Learning Notes

The Science of Data

Statistics is the science of collecting, organising, analysing, interpreting, and presenting data to make informed decisions and draw conclusions under uncertainty.

Early Beginnings

• Babylonians collected census data ~3000 BCE
• Egyptians used data for pyramid construction planning
• Romans conducted systematic population censuses
• India: Arthashastra of Kautilya mentions data collection

Must-Know Terms

• Population (N): Complete set of all items of interest
• Sample (n): Subset of population actually studied
• Parameter: Numerical measure of a population (μ, σ, π)
• Statistic: Numerical measure of a sample (x̄, s, p̂)
• Variable: A measurable characteristic that varies

Original Data (First-hand)

• Direct personal observation
• Questionnaires & structured surveys
• Interviews (direct/indirect methods)
• Experimental data from controlled studies
• Registration systems (births, deaths, marriages)

The "Centre" of Data

A single value representing the typical or central value in a dataset. The three primary measures are Mean, Median, and Mode, each optimal under different data conditions.

Quantifying Variability

Dispersion measures the spread or variability in a dataset. Two datasets can have the same mean but vastly different spreads — dispersion captures this critical difference.

Relative Change Measure

An index number measures the relative change in a variable (or group) compared to a base period. Expressed as a percentage relative to the base (base period = 100). Used to track changes over time.

Data Over Time

A time series is a sequence of data points collected at successive, equally-spaced time intervals. Goal: identify patterns, decompose components, and forecast future values.

Measuring Association

Correlation measures the strength and direction of the linear relationship between two variables. The Pearson correlation coefficient r ranges from −1 to +1.

Line of Best Fit

Regression establishes a mathematical relationship to predict the value of a dependent variable (Y) from an independent variable (X) using the principle of Ordinary Least Squares (OLS).

Non-numerical Characteristics

Attributes are qualitative characteristics (literacy, colour, gender, disease) that are categorised rather than measured. Analysis counts classes and tests association between categories.

Asymmetry of Distribution

• Symmetric (Sk=0): Mean = Median = Mode
• Positive skew (+): Mean > Median > Mode — right tail longer
• Negative skew (−): Mean < Median < Mode — left tail longer

Joint Distribution of (X, Y)

A bivariate distribution shows the joint frequency distribution of two variables simultaneously — revealing their individual behaviour AND their joint patterns and dependence structure.

Sets & Notation

• Set: A well-defined collection of distinct objects
• Roster: A = {1, 2, 3}; Set-builder: {x : x < 4, x ∈ ℕ}
• Universal set (Ω): Contains all elements under study
• Empty set (∅): No elements; ∅ ⊆ every set

Experiment, Outcomes, Events

• Random experiment: Outcome not predictable with certainty
• Sample space (S): Set of ALL possible outcomes
• Event (A): A subset of the sample space
• Mutually exclusive: A ∩ B = ∅
• Exhaustive: A ∪ B = S

Multiplication Rule

If task 1 can be done in m ways and task 2 in n ways, then both can be done in m × n ways. Extended to k tasks: m₁ × m₂ × … × mₖ.

Probability Given Information

P(A|B) = the probability of A given that B has already occurred. We restrict the sample space to B and measure A within it. This is the "updated" probability with new information.

Mapping Outcomes to Numbers

X: S → ℝ assigns a real number to each sample point. Capital X = the RV (function); lowercase x = the value it takes. Converts non-numeric experiments into numbers for analysis.

Single Trial — Success/Failure

• One trial, two outcomes: 1 (success) with prob p, 0 with prob (1−p)
• E(X) = p; Var(X) = p(1−p)
• Building block for Binomial

The Bell Curve — Most Important

• Symmetric about mean μ; inflection points at μ±σ
• 68-95-99.7 rule for 1σ, 2σ, 3σ from mean
• Standard Normal Z ~ N(0,1): Z = (X−μ)/σ
• Central Limit Theorem: sample means → Normal

The Population Model

Y = β₀ + β₁X + ε. We model the linear relationship between a response Y (dependent) and a predictor X (independent), where ε is random error. We estimate β₀ & β₁ from sample data.

SST = SSR + SSE

• SST: Total sum of squares = Σ(Yᵢ−ȳ)²
• SSR: Regression SS = Σ(Ŷᵢ−ȳ)² (explained by model)
• SSE: Error SS = Σ(Yᵢ−Ŷᵢ)² (unexplained/residual)
• R² = SSR/SST — proportion of variance explained

Is X a Significant Predictor?

• H₀: β₁ = 0 (X has no linear effect on Y)
• H₁: β₁ ≠ 0 (X is a significant predictor)
• t = b₁ / SE(b₁) ~ t(n−2) under H₀
• Reject H₀ if |t| > t_(α/2, n−2)

Checking Model Assumptions

Residuals eᵢ = Yᵢ − Ŷᵢ carry information about assumption violations. Always plot residuals before trusting inference. A good model has residuals that look like random noise.

Large Residuals

A point with a large studentised residual |rᵢ| > 2 or 3. Outliers in Y can inflate MSE and distort regression estimates. Check if real or data error.

k Predictors

Y = β₀ + β₁X₁ + β₂X₂ + … + βₖXₖ + ε. Each βⱼ is the partial effect of Xⱼ on Y, holding all other predictors constant. Estimated by matrix algebra: b = (X'X)⁻¹X'Y.

Definition

Econometrics = Economics + Metrics. It applies statistical and mathematical methods to quantify economic relationships, test economic theories, and forecast future economic activity. Gujarati defines it as the "quantitative analysis of actual economic phenomena."

CLRM Assumptions (Gujarati)

• A1: Linear in parameters — model is linear in β (not necessarily in X)
• A2: Fixed X values — X is non-stochastic (or fixed in repeated sampling)
• A3: Zero mean error — E(εᵢ) = 0
• A4: Homoscedasticity — Var(εᵢ) = σ² (constant)
• A5: No autocorrelation — Cov(εᵢ, εⱼ) = 0, i≠j
• A6: X non-stochastic — Cov(εᵢ, Xᵢ) = 0
• A7: n > k — more observations than parameters
• A8: Variability in X — Var(X) ≠ 0
• A9: No perfect multicollinearity — no exact linear relation among Xs
• A10: Normality of ε — εᵢ ~ N(0, σ²)

Minimise Squared Errors

OLS chooses β̂ to minimise SSE = Σeᵢ² = Σ(Yᵢ − Ŷᵢ)². The "squaring" penalises large errors more — like a strict teacher who really hates big mistakes more than small ones! 😄 The solution is unique and closed-form.

Correlated Predictors

Multicollinearity occurs when two or more predictor variables are highly correlated with each other. Perfect multicollinearity = exact linear relationship (OLS breaks down entirely). Near-perfect = high but not perfect correlation (OLS works but gives unreliable estimates).

Non-constant Error Variance

Heteroscedasticity means Var(εᵢ) = σᵢ² — the variance of the error term is NOT constant across observations. It changes with one or more predictors. Very common in cross-sectional data (individuals, firms, countries with very different sizes).

Correlated Error Terms

Autocorrelation (serial correlation) means Cov(εᵢ, εⱼ) ≠ 0 for i≠j. Violations of assumption A5. Most common in time series data (monthly GDP, daily stock prices, annual inflation). Positive autocorrelation is most common — errors persist in the same direction.

Using the Wrong Model

Specification errors arise when the model is incorrectly specified — wrong variables, wrong functional form, or wrong structural assumptions. The most dangerous error in econometrics!

Binary Indicator Variables

For a qualitative variable with m categories, we include m−1 dummy variables (omit one — the "base" or "reference" category). Including all m dummies causes perfect multicollinearity — the dummy variable trap!

Bidirectional Causality

In many economic situations, variables determine each other simultaneously. Supply & demand: price determines quantity demanded AND quantity supplied determines price. This simultaneity causes OLS to be biased and inconsistent — the "simultaneity bias."

The Key Concept in Time Series

A time series is weakly stationary if its mean, variance, and autocovariances are constant over time (don't depend on t). Most economic time series (GDP, prices, exchange rates) are NON-stationary — they have trends and drifts.

Meaning & Scope

Multivariate Analysis (MVA) refers to statistical techniques for analysing data with p ≥ 2 variables measured on each observation. Goal: understand the joint behaviour, interdependencies, and structure of these variables simultaneously — not one at a time.

Ordinary Geometric Distance

The familiar straight-line distance between two points x and y in p-dimensional space: d(x,y) = √[Σᵢ(xᵢ−yᵢ)²]. Simple but has a critical flaw: it treats all variables equally regardless of their scale or correlation. A variable measured in kilometres swamps one measured in centimetres!

Eigenvalue Decomposition

Every symmetric positive definite matrix A can be decomposed as: A = PΛP' where P = matrix of eigenvectors (orthonormal columns) and Λ = diagonal matrix of eigenvalues λ₁ ≥ λ₂ ≥ … ≥ λₚ > 0. The eigenvectors give the "principal directions" of the data; eigenvalues give the "lengths" in those directions. Foundation of PCA!

The Multivariate Analogue of Variance

For a p-dimensional random vector X, the covariance matrix Σ (p×p) captures ALL pairwise variances and covariances: σᵢᵢ = Var(Xᵢ) on diagonal; σᵢⱼ = Cov(Xᵢ,Xⱼ) off-diagonal. Σ is symmetric and positive (semi)definite. The sample version S = (n−1)⁻¹Σᵢ(xᵢ−x̄)(xᵢ−x̄)' is the unbiased estimator.

Nₚ(μ, Σ)

A p-dimensional random vector X follows a multivariate normal distribution Nₚ(μ,Σ) if every linear combination a'X is (univariate) normal for any non-zero vector a. Parameters: mean vector μ (p×1) — location; covariance matrix Σ (p×p) — shape and spread. The MVN is completely characterised by just these two parameters!

Sample Mean Vector

The MLE of the mean vector μ is simply the sample mean vector x̄ = (1/n)Σᵢxᵢ. It is unbiased E(x̄)=μ and its sampling distribution is: x̄ ~ Nₚ(μ, Σ/n). Larger n → smaller variance of x̄ → more precise estimate. Intuition: just average each variable separately.

Univariate Marginal Normality

• Plot histogram and Q-Q plot for each variable separately
• Shapiro-Wilk or Kolmogorov-Smirnov test for each Xⱼ
• Check for skewness and kurtosis near 0 and 3 respectively
• Warning: All marginals normal ≠ joint MVN! This is necessary but NOT sufficient

Multivariate Analogue of χ²

If X₁,…,Xₙ are iid Nₚ(0,Σ), then the matrix W = Σᵢ XᵢXᵢ' ~ Wₚ(n,Σ) follows a Wishart distribution with n degrees of freedom and scale matrix Σ. The sample covariance matrix satisfies: (n−1)S ~ Wₚ(n−1,Σ). It is the matrix generalisation of the chi-square distribution — just as s² has a chi-square distribution in univariate normal, S has a Wishart distribution!

Multivariate t-Test

Tests H₀: μ = μ₀ (mean vector equals a specified vector). Hotelling's T² = n(x̄−μ₀)'S⁻¹(x̄−μ₀). This is the multivariate generalisation of the one-sample t-test. Under H₀: [(n−p)/p(n−1)]·T² ~ Fₚ,ₙ₋ₚ. Reject H₀ if this exceeds Fₐ(p,n−p). The TWO-SAMPLE version tests H₀: μ₁=μ₂ using the pooled covariance matrix.

Multiple Y, Multiple X

Multivariate multiple regression has multiple response variables Y (n×m matrix) AND multiple predictors X (n×(k+1) matrix). Model: Y = XB + E where B (k+1)×m is the coefficient matrix and E is n×m error matrix. Each column of Y is a separate response; they share the same predictors X.

Finding Maximum Variance Directions

PCA transforms p correlated variables into p uncorrelated Principal Components (PCs) that are linear combinations of the originals. PC1 captures maximum variance; PC2 captures maximum of remaining variance orthogonal to PC1; and so on. Goal: represent data in fewer dimensions with minimal information loss.

Beyond Uncorrelated — Finding Independence

ICA decomposes X = AS + noise where S are statistically independent source signals and A is the mixing matrix. Goal: estimate A and recover S. Unlike PCA (finds uncorrelated components), ICA finds components that are statistically independent — a much stronger condition. Non-Gaussian sources are required!

Latent Factor Model

Factor Analysis (FA) models p observed variables as linear combinations of m << p latent (unobservable) common factors F plus unique factors: X = μ + LF + ε. L is the (p×m) loading matrix; F are m common factors; ε are p unique (specific) factors. Goal: interpret the common factors as meaningful latent constructs.

Grouping Without Labels

Cluster analysis partitions n observations into g groups (clusters) such that observations within a cluster are similar and observations between clusters are dissimilar. It is unsupervised — no predefined groups or labels. Goal: discover natural structure in data.

Supervised Group Separation

Discriminant Analysis has TWO goals: (1) Description: find linear combinations of variables (discriminant functions) that best separate g known groups; (2) Classification: build a rule to assign future observations to one of the g groups. Unlike cluster analysis: group memberships are KNOWN for the training data.

The Complete Set

• Population (N): All items of interest — fixed but usually unobservable
• Parameter: μ, σ², π — numerical summaries of the population, FIXED but UNKNOWN
• Almost never observe the whole population — too large, costly, or destructive

Exact Result (Any n)

If X₁,…,Xₙ iid N(μ,σ²), then x̄ ~ N(μ, σ²/n) exactly for any n. Standardise: Z = (x̄−μ)/(σ/√n) ~ N(0,1). When σ unknown, replace with s → T = (x̄−μ)/(s/√n) ~ t(n−1).

Most Important Theorem in Statistics

Let X₁,…,Xₙ be iid with mean μ and finite variance σ². Then as n→∞: √n(x̄−μ)/σ →_d N(0,1) regardless of the population distribution shape. For large n: x̄ ≈ N(μ, σ²/n). This is why the normal distribution appears everywhere!

Sum of Squared Normals

• χ²(k) = Z₁²+…+Zₖ² where Zᵢ iid N(0,1)
• Mean=k; Var=2k; Right-skewed; always ≥ 0
• Sampling dist of variance: (n−1)s²/σ² ~ χ²(n−1)
• ⚠ Requires population normality — sensitive to departures!

For Binary Outcomes

p̂ = X/n where X~Binomial(n,p). E(p̂)=p (unbiased); Var(p̂)=p(1−p)/n. By CLT: p̂ ≈ N(p, p(1−p)/n) when np≥10 AND n(1−p)≥10. Standard error: SE(p̂) = √[p(1−p)/n].

Comparing g Group Means

H₀: μ₁=μ₂=…=μg vs H₁: at least one μᵢ differs. Partitions total variation: SST = SSA + SSE. If between-group (MSA) >> within-group (MSE), groups differ. F = MSA/MSE ~ F(g−1, N−g) under H₀.

Independence · Equal Variance · Normality

• Independence: All observations independent — ensured by randomisation
• Homoscedasticity: σ₁²=…=σg² — test with Levene's test (robust) or Bartlett's (sensitive to non-normality)
• Normality: Residuals eᵢⱼ=yᵢⱼ−ȳᵢ. ~ N(0,σ²) — check Q-Q plot or Shapiro-Wilk

Model & Decomposition

Tests: (1) Main effect of A; (2) Main effect of B; (3) Interaction A×B. SST = SSA + SSB + SSAB + SSE. Interaction is most interesting — does the effect of A depend on the level of B? Plot interaction plots: parallel lines = no interaction; crossing lines = interaction present.

Completely Randomised Design

Treatments randomly assigned to all units with no restrictions. Simplest design — use when units are homogeneous. Analysis: one-way ANOVA. df_error = N−t. Disadvantage: if units are heterogeneous, MSE will be large and F-test will be weak.

Setting Up the Test

• H₀: Status quo / no effect — assumed true by default
• H₁: What we're trying to demonstrate
• Simple: Completely specifies distribution (μ=5)
• Composite: Specifies a class (μ>5)
• One vs two-sided: H₁: μ>μ₀ vs H₁: μ≠μ₀

Most Powerful Test for Simple H

For H₀:θ=θ₀ vs H₁:θ=θ₁ (both simple), the Most Powerful (MP) test at level α rejects H₀ when Λ(x) = L(θ₁)/L(θ₀) > k. The N-P Lemma derives the optimal rejection region from the likelihood ratio — no guessing needed. For Gaussian data, this recovers the z-test as optimal.

Correct Definition

p-value = P(T ≥ t_obs | H₀) = probability of data as extreme or more extreme than observed, assuming H₀ true. Small p → data surprising under H₀ → evidence against H₀. It is a continuous measure of evidence, NOT a binary pass/fail.

Sampling Vocabulary

• Sampling frame: List of all N population units — must be complete and up-to-date
• Sampling unit: The unit selected at each draw
• Inclusion probability πᵢ: Probability unit i is selected
• Design effect (DEFF): Ratio of actual variance to SRS variance

With vs Without Replacement

• SRSWOR: Each unit selected at most once — more common; smaller variance
• SRSWR: Units can repeat — simpler theory; larger variance
• Finite population correction (FPC) = (1−f) = (1−n/N) — matters when n/N > 0.05

Divide & Sample

Divide population into L non-overlapping strata; take SRS within each stratum. Why? Reduces variance by removing between-stratum variation from the error. Always more efficient than SRS if strata are internally homogeneous.

Every kth Unit

• k = N/n (sampling interval); select random start r ∈ {1,…,k}; then r, r+k, r+2k, …
• Very easy to implement — just a list and arithmetic
• Efficient when list is in random order (≈SRS)
• ⚠ Periodic pattern in list + periodic k = biased disaster!

Using a Correlated Auxiliary Variable

If auxiliary variable X (known population total X̄) is highly correlated with Y: ȳ_R = R̂·X̄ where R̂=ȳ/x̄. Biased but often much lower MSE than ȳ. Best when ratio Y/X is more constant than Y itself — e.g., estimating crop yield per hectare.

Probability Proportional to Size

Select PSUs with probability proportional to a size measure (number of households, land area). Larger clusters have higher selection probability. Combined with equal-probability sub-sampling within PSUs → self-weighting sample. Used in almost all national surveys.

Binomial, Multinomial & Poisson

• Binomial(n,π): n independent trials, count successes. Foundation for proportions.
• Multinomial(n; π₁,…,πk): n trials, k categories. Joint distribution of cell counts.
• Poisson(μ): Independent cell counts — used in log-linear models

Cross-Tabulation

An r×c table cross-classifies n observations by two categorical variables (r rows, c columns). Cell count nᵢⱼ = observations in row i, column j. Marginal totals: nᵢ₊ (row), n₊ⱼ (column). Test: are the two variables independent?

Most Important Association Measure

OR = (n₁₁·n₂₂)/(n₁₂·n₂₁) = (odds of outcome in group 1)/(odds in group 2). OR=1 means no association. OR>1 means higher odds in group 1. OR is the natural parameter for logistic regression and case-control studies. Does not depend on marginal totals — unlike RR.

Logit Link Function

For binary Y∈{0,1}: log[π/(1−π)] = β₀ + β₁X₁ + … + βₖXₖ where π = P(Y=1|X). The logit link ensures predicted probabilities ∈ (0,1). Estimated by Maximum Likelihood Estimation (MLE), not OLS. Iteratively Reweighted Least Squares (IRLS) algorithm.

Modelling Cell Counts

Log-linear models treat cell counts as Poisson: ln(μᵢⱼ) = λ + λᵢᴬ + λⱼᴮ + λᵢⱼᴬᴮ. All variables are response variables — no distinction between X and Y. Especially useful for 3+ way tables to model partial and conditional independence structures.

Concordant & Discordant Pairs

• Concordant pair: Both variables rank same direction
• Discordant pair: Variables rank opposite directions
• Gamma γ: (C−D)/(C+D) ∈ [−1,1] — ignores ties
• Kendall's τb: Accounts for ties — preferred
• Spearman's ρ: Correlation of ranks

Systematic Inquiry

Research is a systematic, controlled, empirical investigation of natural phenomena guided by theory and hypotheses about the relationship between variables. It is not just "searching the web" — it requires rigour, replicability, and transparency.

Positivism, Interpretivism & Pragmatism

• Positivism: Objective reality exists; can be measured; deductive; quantitative
• Interpretivism: Reality is socially constructed; context matters; inductive; qualitative
• Pragmatism: "Whatever works" — mixed methods; research question drives method choice
• Most statistics students work within a positivist paradigm

Why Review the Literature?

• Identifies what is already known — avoid duplicating work
• Locates gaps your research fills
• Provides theoretical framework and conceptual models
• Guides appropriate methodology and instruments
• Databases: PubMed, Web of Science, Scopus, Google Scholar, JSTOR

Nominal · Ordinal · Interval · Ratio

• Nominal: Gender, religion, blood type — categories only; mode appropriate
• Ordinal: Education level, satisfaction rating — ranked; median appropriate
• Interval: Temperature, IQ — equal intervals, no true zero; mean appropriate
• Ratio: Income, weight, height — true zero; all measures appropriate

Are We Measuring What We Intend?

• Content validity: Items cover the full domain
• Construct validity: Measures the theoretical construct (convergent + discriminant)
• Criterion validity: Correlates with gold standard (concurrent + predictive)
• Internal validity: Study design allows causal inference (no confounding)
• External validity: Results generalise to other populations/settings

Survey, Interview, Observation, Secondary

• Self-administered survey: Cheap, large scale, no interviewer bias — but low response rate
• Face-to-face interview: High response, complex questions possible — expensive, interviewer bias
• Telephone/CATI: Moderate cost, fast — coverage bias (no mobile?)
• CAPI: Computer-assisted personal interview — error reduction, skip patterns automated
• Secondary data: HIES, DHS, census, administrative records